Preparing Data For The Data Lake

Data preparation is increasingly becoming one of the biggest challenges in processing big data. While recent tools such as Tamer and Trifacta address the problem of integrating and cleaning the datasets as they come in, preparing these datasets for efficient processing over a variety of query workloads is still challenging. In this talk, I will discuss these challenges and describe our tool which allows for fine-grained data preparation, via a data preparation plan, and efficiently runs this plan while uploading the data to HDFS.